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ABSTRACT 

We calculate the statistical clustering of Lyman-break galaxies predicted in a 
selection of currently-fashionable structure formation scenarios. These mod- 
els are all based on the cold dark matter model, but vary in the amount of 
dark matter, the initial perturbation spectrum, the background cosmology and 
in the presence or absence of a cosmological constant term. If Lyman-break 
galaxies form as a result of hierarchical merging, the amplitude of clustering 
depends quite sensitively on the minimum halo mass that can host such a 
galaxy. Interpretation of the recent observations by Giavalisco et al. (1998) 
would therefore be considerably clarified by a direct determination of the rel- 
evant halo properties. For a typical halo mass around 10 n /i -1 M the obser- 
vations do not discriminate strongly between cosmological models, but if the 
appropriate mass is larger, say 10 12 /i _1 M Q (which seems likely on theoretical 
grounds), then the data strongly favour models with a low matter-density. 

Key words: cosmology: theory - cosmology: observations - large-scale struc- 
ture of Universe - galaxies: formation - galaxies: evolution - galaxies: haloes 



1 INTRODUCTION 

Developments in observational techniques have recently 
led to an explosion of interest in the properties of cosmo- 
logical objects at such high redshifts that the lookback 
time at which they are seen is a considerable fraction 
of the age of the Universe. Such objects therefore fur- 
nish the opportunity to probe directly the evolution of 
galaxy clustering, bridging the gap between local obser- 
vations of large-scale structure (i.e. observations with 
lookback times that are small compared to the age of 
the Universe) and observations of the cosmic microwave 
background radiation (where the lookback time is vir- 
tually equal to the age of the Universe). 

In Matarrese et al. (1997; hereafter Paper I) and 
Moscardini et al. (1998; hereafter Paper II), we dis- 
cussed high-redshift clustering phenomena from a theo- 
retical perspective in order to provide a general frame- 
work within which these high-redshift phenomena can 
be interpreted. The formalism we derived can be used to 
make detailed predictions of statistical measures of clus- 
tering in specific cosmological scenarios and also makes 
explicit the main sources of theoretical uncertainty in 
these predictions. This allows one to make a realistic 
assessment of how models of structure formation fare 
in the face of results from particular observational pro- 
grammes. In Paper II we applied this approach to a con- 



frontation of different cosmological models with various 
galaxy clustering data, including a brief discussion of 
the Lyman-break galaxies (LBGs) presented in Steidel 
et al. (1998). 

The properties of LBGs are presently undergoing a 
great deal of scrutiny because of the opportunities they 
present to study systematically the spatial clustering of 
galaxies at extremely high redshifts. It is the purpose 
of this paper to deploy the techniques of Papers I & II 
in a systematic comparison of a selection of currently 
popular structure formation models with recent results 
on the angular correlations of LBGs presented by Gi- 
avalisco et al. (1998). 

LBGs are identified by using an efficient photomet- 
ric technique that allows the identification of candidate 
high-redshift objects through the shifting of their Ly- 
man limit cutoff into a particular colour filter (Stei- 
del & Hamilton 1993). Recently Steidel et al. (1996, 
1998) began a survey for z ~ 3 galaxies using this 
Lyman-break technique by observing five different fields 
altogether covering approximately 700 arcmin 2 . They 
found 871 candidates with magnitude 1Z < 25.5. The 
subsequent spectroscopic identification for 376 galaxies 
showed that approximately 90 per cent of the objects 
have 2.6 < z < 3.4, with a median redshift z — 3.04 and 
a small r.m.s. (a z — 0.27). 

The interpretation of LBGs is not entirely straight- 
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forward. They are certainly small, but are highly lu- 
minous objects. Moreover, interpretating their strong 
interstellar absorption lines as essentially due to grav- 
itational motions, their masses seem to be similar to 
those of local bright galaxies (Giavalisco, Steidel & Mac- 
chetto 1996). This favours the interpretation that they 
comprise a massive halo within which the formation of a 
galaxy is in progress (see e.g. Baugh et al. 1998) . The al- 
ternative possibility that LBGs are relatively low-mass 
(~ 1O 1O M0) objects with an intense starburst activity 
has been discussed by Somerville (1997). 

The problem of the LBGs, and in particular the 
probability that a strong concentration at z ~ 3, similar 
to that reported by Steidel et al. (1998), arises in par- 
ticular cosmological scenarios, have been discussed by 
various authors by using both analytical and N-body 
techniques (Mo & Fukugita 1996; Jing & Suto 1998; 
Governato et al. 1998; Bagla 1997a; Wechsler et al. 1997; 
Peacock et al. 1998; Paper II). The resulting picture 
seems to indicate that at z ~ 3 LBGs are strongly bi- 
ased because they lie at the highest peaks of the density 
field (Kaiser 1984). They are therefore expected to be 
strongly clustered in all currently-fashionable cosmolog- 
ical models. If this interpretation is correct, LBGs would 
be the progenitors of the population of massive ellipti- 
cals (Steidel et al. 1998) or cluster galaxies we see at 
2 = (Governato et al. 1998). 

The plan of this Letter is as follows. In Section 2 we 
describe our formalism to study the clustering at high 
redshift. In Section 3 we present the cosmological mod- 
els used in the following analysis. The predictions of the 
correlation functions of the LBGs are shown in Section 
4. The final discussion and conclusions are presented in 
Section 5. 



2 CLUSTERING AT HIGH REDSHIFT 

In Papers I & II we showed that the observed spatial 
correlation function £ b s in a given redshift interval Z 
is an appropriate weighted average of the mass autocor- 
relation function £ with the mean number of objects M 
and effective bias factor 6 c ff, defined below in equation 
(B), in that range: 

£obs(r) = N~ 2 J dz 1 dz2 

x AT(zi) N(z 2 ) &eff(zi) b cB (z 2 ) £{r,z) , (1) 

where N = J z dz'Af(z') and z is an intermediate red- 
shift between z\ and 22; see Papers I & II for details. 

The factor of b e ff which appears in equation ([j]) 
is a consequence of our lack of understanding of the 
details of the galaxy formation process and the conse- 
quently uncertain relationship between fluctuations in 
matter density 8 m and galaxy number-density S n . We 
assume that objects with given intrinsic properties (such 
as mass M) and at different redshifts z can have differ- 
ent bias parameters, which we call b(M, z). For each set 
of objects, however, the bias is assumed still to be lin- 
ear; it is also local (e.g. Coles 1993), in the sense that the 
propensity of galaxies to form at a given spatial location 
x depends only on the matter density at that point: 



<5 n (x; M, z) ~ b(M, 2)<5 m (x, 2); (2) 

so that no environmental or cooperative effects in galaxy 
formation (e.g. Babul & White 1991; Bower et al. 1993) 
are permitted. If we assume such a bias between the 
galaxy and mass fluctuations, the effective bias factor 
b e « (2) which appears in equation (|l|) can be expressed as 
a suitable average of the "monochromatic" bias b(M, 2) 
(i.e. the bias factor of each single object): 

b cS {z)=M{z)- 1 j d\nM' b{M',z) Af(z,M') . (3) 
J M 

In principle the variable M (and its range M) stands 
for any intrinsic properties of the object in question 
(e.g. mass, luminosity, etc.) on which the selection of 
the object into an observational sample might depend. 
From here on, however, we shall assume that all such 
properties can be reduced to a dependence on the mass 
of the halo within which the object (galaxy) forms. 
This assumption is to some extent debatable (see, e.g., 
Kauffmann, Nusser & Steinmetz 1997; Roukema et al. 
1997) but seems a reasonable starting point in the lim- 
ited range of redshifts relevant to the LBG population. 
Moreover, Haehnelt, Natarajan & Rees (1997) have re- 
cently shown that a good fit to the LBG luminosity 
function can be obtained by assuming a linear relation 
between star formation rate and halo mass, which im- 
plies a constant ratio of mass-to-UV- light. Henceforth 
in this study, therefore, M can be taken to stand for 
the mass of the parent halo of the LBG. 

In order to predict the clustering properties of ob- 
jects as a function of 2 we need to understand how the 
relationship between these objects and the underlying 
mass distribution evolves. In most fashionable models 
of structure formation the growth of structures on a 
given mass scale is driven by the hierarchical merging of 
sub-units. One begins by calculating the bias parameter 
b(M, 2) for haloes of mass M and 'formation redshift' 
2/ at redshift z < Zf in a given cosmological model. The 
result is 

where a\j is the linear mass-variance averaged over the 
scale M extrapolated to the present time (z — 0) and 
5c is the critical linear overdensity for spherical collapse 
[6c = const = 1.686 in the Einstein-de Sitter case, while 
it depends slightly on 2 for more general cosmologies 
(Lilje 1992)]. The above expression originally appeared 
(in a slightly different form) as equation (6) of Cole 
& Kaiser (1989). It was then later discussed by Mo & 
White (1996) who also compared it with the results of 
numerical experiments which showed good agreement 
with the simple theoretical form. The general non-linear 
relation between the halo and the mass density contrast 
has been recently obtained by Catelan et al. (1998), by 
solving the continuity equation for dark matter haloes. 
A complementary study by Bagla (1997b) has further 
explored the clustering of haloes using numerical exper- 
iments; see also Ogawa, Roukema & Yamashita (1997). 

As in Paper I, we can estimate the effective bias 
by assuming that the objects observed in a given sur- 
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vey represent all haloes exceeding a certain cutoff mass 
M m in at any particular redshift. In other words, we 
assume that there is a selection function <j>(z, M) = 
Q(M — M m i n ) at any z, where &(•) is the Heaviside step 
function. In this way, by modelling the linear bias at red- 
shift z for haloes of mass M as in equation (^) and by 
weighting it with the theoretical mass-function n(z, M) 
which we can self-consistently calculate using the Press- 
Schechter (1974) theory, we can obtain the behaviour of 
b e a(z) directly. If we were to assume that rapid merging 
continued up to the present then this model (that in Pa- 
pers I & II we called the merging model) is completely 
defined by the initial amplitude of primordial density 
fluctuations, because that determines the scale of non- 
linearity at each epoch z. This would mean that the 
parameter M m i n is fixed by requiring the present pop- 
ulation of galaxies to have been entirely produced by 
a merger-driven hierarchy. Accordingly the present-day 
value of bcff(z = 0), which can be extracted by com- 
paring local measurements with the mass fluctuations 
predicted in a given model, determines Mmin- 

We feel, however, that the assumption that one can 
compare the clustering of present-day galaxies directly 
with that of LBGs (which are selected in an entirely dif- 
ferent way) is rather unsafe. Moreover, it may well be 
the case that instantaneous merging is not a good ap- 
proximation for the later stages of clustering evolution. 
It is more reasonable therefore to regard M m i n as a free 
parameter and not attempt to relate the properties of 
LBGs to local galaxies. In Paper I & II we called the 
model obtained by letting M m i n be a free parameter the 
transient model, because the original theoretical moti- 
vation for it was the case of high-z QSOs which have no 
obvious counterpart among the local galaxy population. 
It should, however, also be a good model for LBGs. The 
choice of minimum halo mass for the LBG case is not 
obvious, so in the following we give results for two rep- 
resentative cases (10 11 and 1O 12 /i -1 M0). The higher of 
these values is favoured by the properties of absorption 
lines of these objects (Steidel et al. 1998; see, however, 
Somerville 1997). 

The computation of clustering properties using 
equation (1) is completed by the specification of the 
matter covariance function and its evolution with z, i.e. 
£(r,z). As in Papers I & II we use a method based on 
the original suggestion by Hamilton et al. (1991) and 
developed by Peacock & Dodds (1994), Jain, Mo & 
White (1995) and Peacock & Dodds (1996) to calcu- 
late the evolution of perturbations into the non-linear 
regime. This technique also takes account of different 
background cosmologies, possible contributions from a 
cosmological constant and can be applied to a variety 
of initial perturbation spectra. 



3 A SUITE OF TRIAL COSMOLOGIES 

In this paper we consider a set of cosmological models 
which can all be regarded as variations on the basic cold 
dark matter (CDM) scenario. Although the Standard 
CDM model (SCDM) is no longer regarded as a good fit 
to observations of galaxy clustering and the microwave 



background, there are several alternatives with many of 
the same basic features but with differences in detail. In 
a general way, the initial (linear regime) power spectrum 
for all these models, which provides the initial condi- 
tions for the clustering evolution calculations discussed 
above, can be represented by Pi in (&,0) = P k n T 2 (k), 
where we use the fitting formula of the CDM trans- 
fer function T(k) as given by Bardeen et al. (1986). 
To fix the amplitude of the power spectrum (generally 
parametrised in terms of as, the r.m.s. fluctuation am- 
plitude inside a sphere of 8/i _1 Mpc) we either attempt 
to fit the present-day cluster abundance or the level of 
fluctuations observed by COBE (Bunn & White 1997). 

We will consider the following specific models: the 
SCDM model, as reference model, with n = 1 and a 
normalization consistent with the COBE data (as = 
1.22); a different version of the SCDM model (here- 
after called SCDMci) with a reduced normalization 
(as = 0.52) producing a cluster abundance in agreement 
with the observational data (Eke, Cole & Frenk 1996; 
see also Viana & Liddle 1996); a COBE-normalized 
tilted model (hereafter TCDM; see e.g. Lucchin & 
Matarrese 1985) with n = 0.8, as = 0.72 and high bary- 
onic content (10 per cent; see White et al. 1996; Gheller, 
Pantano & Moscardini 1998); a different version of the 
previous model, hereafter TCDMgh', with a reduced 
normalization of the scalar perturbations (as = 0.51) 
taking into account the possible production of gravita- 
tional waves, as predicted by some inflationary theories 
(e.g. Lucchin, Matarrese & Mollerach 1992; Lidsey & 
Coles 1992); an open CDM model (hereafter OCDM), 
with a matter density parameter flom = 0.4, a Hubble 
parameter h = 0.65 and COBE-normalized (as = 0.64); 
a low-density CDM model (hereafter ACDM) always 
with flom = 0.4 but with a flat geometry provided by 
the cosmological constant, with h = 0.65 and COBE- 
normalized (as = 1.07). 



4 PREDICTIONS AND OBSERVATIONS 

We presented calculations of the spatial correlations of 
LBGs in Paper II. Although the recent paper by Gi- 
avalisco et al. (1998) included an updated form of the 
redshift distribution, we have verified that this does not 
alter the predictions significantly. Observational esti- 
mates of the spatial two-point function are not yet avail- 
able but Giavalisco et al. (1998) have presented results 
for the angular correlations of LBGs. In our previous pa- 
pers we showed that, in the small-angle approximation, 
the observed two-point angular correlation function u; b s 
can be expressed as: 

Wobs(tf) = N~ 2 J dzG(z) N 2 (z)b 2 cS (z) 

/oo 
du Z[r(u,fi,z),z] , (5) 
-oo 

where r(u,-ff,z) = ao-^/it 2 + x 2 (z)'d 2 and x(z) depends 
on the background cosmology; see Paper II for details. 
Note that the redshift distribution N(z) is an observed 
quantity for a given survey so we do not need to spec- 



© 0000 RAS, MNRAS 000, 000-000 



4 Coles et al. 




6 (arcsec) 

Figure 1. Theoretical prediction in different cosmological 
models for the angular correlation function of the Lyman- 
break galaxies. The adopted redshift distribution and the 
correlation data are taken from Giavalisco et al. (1998). Open 
and filled squares (with la errorbars) refer to two different 
estimators of the angular correlation, PB and LS respec- 
tively (see Giavalisco et al. 1998 for a discussion). Two differ- 
ent minimum masses are used to compute the effective bias: 
lO 11 fe" 1 M (top panel) and 1O 12 /i" 1 M (bottom panel). 



ify this function theoretically. In the following analysis 
we adopt the same N(z) presented in Giavalisco et al. 
(1998). 

In Figure 1 we compare the predictions of w bs($) 
for each of the models presented in the previous sec- 
tion and in each case for two choices of the minimum 
halo mass with the observational results obtained by Gi- 
avalisco et al. (1998) using the 871 LBG candidates. It 
can be seen that most models are consistent with the ob- 
servations if the minimum halo mass is relatively small 
(10 11 h^ 1 Mq), whereas the larger choice is much more 
strongly constrained. Indeed in the latter case only the 
two low-density models ACDM and OCDM are consis- 
tent with observations. The SCDM model does fit the re- 
sults at larger angular scales, but is discrepant on small 
scales. 

Predictions of the projected real-space correlation 
function w ohB can be directly obtained by £ b s (r) as 



1 OOO p 1 1 1 — i — i — i i i i 1 1 1 — i — i — i i i_ 




^ I I I I I I I I I I I I I I I I I I 

0.1 0.5 1 5 10 



r p (h _1 Mpc) 

Figure 2. Theoretical prediction in different cosmological 
models for the projected correlation function of the Lyman- 
break galaxies as a function of the (comoving) separation r p 
(in units of h^ 1 Mpc). The redshift distribution is given by 
Giavalisco et al. (1998). Two different minimum masses are 
used to compute the effective bias: 10 11 h -1 Mq (top panel) 
and 1O 12 A _1 M (bottom panel). 



Wob s (r- p ) =2 drr (r 2 - r 2 p y 1/2 , (6) 

J r p 

where r p is the component of the pair separation perpen- 
dicular to the line of sight. Predictions for the projected 
correlation function are shown in Figure 2. 

Notice that the order of the amplitudes of these 
curves for different models is different from the angu- 
lar correlation function. This is because of the differ- 
ent weighting by redshift and the dependence on back- 
ground cosmology of the formulae. 

One has to be a little cautious about the interpre- 
tation of these results because of the relatively small 
physical scales being probed by the sky correlations ob- 
served. The formula (4) was derived using quasi-linear 
arguments which are not strictly valid on small length 
scales. In particular, one would expect that for spatial 
separations of order half the initial Lagrangian radius 
of the haloes, their correlation function should become 
negative due to exclusion effects (Lacey & Cole 1994; 
Porciani et al. 1998). This problem is not restricted to 
this analysis, but is endemic in studies of this kind (e.g. 
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Baugh et al. 1998). In practical terms it is particularly 
relevant in the case of an Einstein-de Sitter model and 
for a large halo mass (and therefore large Lagrangian 
radius): if M = W 12 h~ 1 M then equation (4) is for- 
mally suspect on length scales subtending an angle of 
about 20 arcseconds at z = 3 if fio = 1. Recent studies 
(e.g. Mo & White 1996) seem to show that the extrapo- 
lation of equation (4) onto small scales is in fairly good 
agreement with numerical experiments that incorporate 
non-linear gravitational effects on small scales that are 
absent from the simple theoretical calculation discussed 
above. We are therefore fairly confident about these pre- 
dictions, given the limitations of the galaxy formation 
models, but this issue should be explored more fully us- 
ing high resolution N-body simulations. 



5 DISCUSSION 

The most important results of this work are shown in 
Figure 1. This shows clearly that the angular cluster- 
ing of LBGs is a potential powerful discriminator be- 
tween cosmological structure formation models. Differ- 
ent models predict clearly different angular correlation 
functions for the LBG distribution function. The most 
significant cause of uncertainty is the minimum halo 
mass for formation of an LBG. If the critical halo mass 
is relatively small, e.g. 10 11 h -1 M@, then present data 



do not clearly discriminate between the suite ot models 



we discuss, un tne otner nana, it tne minimum mass is 
a factor of ten larger then only the ACDM and OCDM 
models are allowed. In such a case the data favour a 
Universe with low matter-density, although they do not 
appear to be strongly sensitive to the presence or ab- 
sence of a vacuum energy density (i.e. a cosmological 
constant term). 

It is clearly important to decide what value of the 
threshold mass is more appropriate for this population 
of objects. Theoretical arguments generally favour the 
higher of the two values we have suggested here (e.g. 
Steidel et al. 1998; Paper II). Observationally, one might 
hope to determine the masses of the LBGs through 
their velocity dispersions. But the likeliest explanation 
of these objects at present seems to be that they form 
within rather massive haloes, but do not yet corre- 
spond to fully-assembled galaxies. Velocity dispersions 
might therefore merely reflect the local velocities of star- 
forming regions, which would lead to an underestimate 
of the halo mass. Such data would therefore need to be 
interpreted within the framework of a complete model 
of galaxy formation and evolution (Baugh et al. 1998; 
Governato et al. 1998). Such models would also estab- 
lish a clearer connection the LBG population and that 
of nearby galaxies at z ~ 0. 

Despite the uncertainties surrounding the nature 
and identity of the LBGs, it is reassuring that their 
clustering properties nevertheless seem to fit within the 
standard gravitational instability scenario. Moreover, 
they also seem to be in line with the emerging consensus 
that we live in a Universe with a matter density which 
is less than the critical density. 
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